Migrate CI to pixi by crusaderky · Pull Request #9276 · dask/distributed

crusaderky · 2026-05-25T23:32:51Z

Follow-up to dask/dask#12389

drastically simplify deployment for developers, particularly when juggling multiple environments
drastically improve reproducibility between CI and desktop
prevent CI from failing overnight - with the exception of the upstream workflows, which retain the old behaviour by design

Functional changes

The Tests workflow is still triggered on schedule in order to detect test flakiness, but is no longer affected by upstream changes.
⚠️ The Tests workflow no longer automatically tracks the dask/dask git tip. ⚠️ If a PR in dask/dask breaks something in dask/distributed CI, you will still notice it in the Upstream workflow.
Added Upstream workflow, which features two runs:
- nightly (nightly wheels and git tip of key dependencies)
- python 3.14 (all latest releases of all dependencies).
Upstream / py314 differs from Tests / py314 as Tests uses the lockfile, meaning dependency versions don't change on their own, whereas Upstream throws out the lockfile, getting all latest and greatest packages as soon as they are released. In other words, Upstream / py314 replicates the behaviour of the old Tests workflow.
Repaired the Conda build workflow, which has been broken for some time. Now Conda build runs on all PRs (takes under 5 minutes).
Run tests on Linux ARM
Tweaked the triggers of some CI jobs for sanity

Follow-ups

Migrate test report to pixi (depends on dask/distributed)
Add pixi.toml to dask/partd and dask/zict (short of a full migration to pixi) and reintroduce their git tips to the nightly environment. This is low priority as these packages barely see any movement nowadays.
Run a new pixi task in CI to build and validate pypi wheels
Use pixi to manage linters
Use pixi to automate docs building locally
Use pixi to run with cupy (no CI; only local)

Known issues

pixi currently has poor support for SCM versioning. There are a few workarounds in this PR.
github does not offer a way to hide line counts in lockfiles. This annoyingly causes bugous line counts (e.g. 51k lines in this PR!) whenever pixi.lock changes. For clarity, this is an issue shared by all package managers for all languages which use a lock file.

github-actions · 2026-05-26T00:28:47Z

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

3 files - 28 3 suites - 28 0s ⏱️ - 10h 29m 6s
4 111 tests + 31 3 919 ✅ - 41 190 💤 + 77 2 ❌ - 4
6 589 runs - 52 631 6 273 ✅ - 50 336 314 💤 - 2 290 2 ❌ - 4

For more details on these failures, see this check.

Results for commit cd9bc86. ± Comparison against base commit d3c7b24.

This pull request removes 56 and adds 87 tests. Note that renamed tests count towards both.

distributed.cli.tests.test_dask_scheduler ‑ test_signal_handling[15]
distributed.cli.tests.test_dask_scheduler ‑ test_signal_handling[2]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_scheduler[15]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_scheduler[2]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_worker[15-Nanny]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_worker[15-Worker]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_worker[2-Nanny]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_worker[2-Worker]
distributed.cli.tests.test_dask_ssh
distributed.cli.tests.test_dask_worker ‑ test_listen_address_ipv6[tcp://[::1]:---nanny]
…

distributed.cli.tests.test_dask_worker.test_listen_address_ipv6[tcp:..[ ‑ 1]:---nanny]
distributed.cli.tests.test_dask_worker.test_listen_address_ipv6[tcp:..[ ‑ 1]:---no-nanny]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape0]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape1]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape2]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape3]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape4]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-u1-shape0]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-u1-shape1]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-u1-shape2]
…

This pull request removes 37 skipped tests and adds 85 skipped tests. Note that renamed tests count towards both.

distributed.cli.tests.test_dask_ssh
distributed.cli.tests.test_dask_worker ‑ test_listen_address_ipv6[tcp://[::1]:---nanny]
distributed.cli.tests.test_dask_worker ‑ test_listen_address_ipv6[tcp://[::1]:---no-nanny]
distributed.dashboard.tests.test_components
distributed.dashboard.tests.test_scheduler_bokeh
distributed.dashboard.tests.test_worker_bokeh
distributed.deploy.tests.test_old_ssh
distributed.deploy.tests.test_ssh
distributed.deploy.tests.test_ssh ‑ test_defer_to_old
distributed.diagnostics.tests.test_cudf_diagnostics
…

distributed.cli.tests.test_dask_worker.test_listen_address_ipv6[tcp:..[ ‑ 1]:---nanny]
distributed.cli.tests.test_dask_worker.test_listen_address_ipv6[tcp:..[ ‑ 1]:---no-nanny]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape0]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape1]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape2]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape3]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-f4-shape4]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-u1-shape0]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-u1-shape1]
distributed.protocol.tests.test_numba ‑ test_serialize_numba[serializers0-C-u1-shape2]
…

This pull request skips 35 and un-skips 6 tests.

distributed.deploy.tests.test_subprocess ‑ test_raise_on_windows
distributed.tests.test_client ‑ test_allow_restrictions
distributed.tests.test_client ‑ test_client_num_fds
distributed.tests.test_client ‑ test_client_replicate_host
distributed.tests.test_client ‑ test_computation_object_code_not_available
distributed.tests.test_client ‑ test_file_descriptors_dont_leak[Nanny]
distributed.tests.test_client ‑ test_file_descriptors_dont_leak[Worker]
distributed.tests.test_client ‑ test_mixed_compression
distributed.tests.test_client ‑ test_nbytes_determines_worker
distributed.tests.test_client ‑ test_performance_report[False]
…

distributed.cli.tests.test_dask_scheduler ‑ test_uvloop
distributed.cli.tests.test_dask_spec ‑ test_uvloop
distributed.cli.tests.test_dask_worker ‑ test_uvloop[--nanny]
distributed.cli.tests.test_dask_worker ‑ test_uvloop[--no-nanny]
distributed.tests.test_config ‑ test_uvloop
distributed.tests.test_dask_collections ‑ test_sparse_arrays

♻️ This comment has been updated with latest results.

crusaderky · 2026-05-26T21:30:23Z

        id: run_tests
        shell: bash -l {0}
-        env:
-          PYTHONFAULTHANDLER: 1


Removed; didn't see a big need for it. I haven't seen segfaults in a very long time.

crusaderky · 2026-05-26T21:32:26Z

-
-    s = sparse.COO([x], data)
-
+    s = sparse.COO(x)


sparse tests were never running before

crusaderky · 2026-05-26T21:35:00Z

+# Optional dependencies
+cytoolz = "=0.11.2"
+lz4 = "=4.3.2"


Unlike in dask/dask, there aren't two separate environments mindeps-non-optional vs. mindeps-optional. Adding them is out of scope for this PR and could be considered in a follow-up.

crusaderky · 2026-05-26T21:37:06Z

+[feature.optional-problematic.target.linux-64.dependencies]
+crick = "*"  # Not available for 3.14, 3.14t, and linux-aarch64
+memray = "*"   # Not available on Windows
+uvloop = "*"  # Not available on Windows


uvloop is new in CI. Its tests did not run before.

crusaderky · 2026-05-26T21:37:46Z

+scipy = "*"
+sparse = "*"
+setproctitle = "*"
+# stacktrace = "*"  # FIXME long-forgotten dependency, must investigate


There is some functionality around stacktrace with broken and forgotten tests.
This deserves its own follow-up.

crusaderky · 2026-05-26T21:39:00Z

+[feature.nightly.dependencies]
+python = "=3.14"
+# See ../dask/continuous_integration/pixi-recipes/README.md
+fsspec = { git = "https://github.com/dask/dask", subdirectory = "continuous_integration/pixi-recipes/fsspec" }
+s3fs = { git = "https://github.com/dask/dask", subdirectory = "continuous_integration/pixi-recipes/s3fs" }
+
+[feature.nightly.pypi-dependencies]
+numpy = { index = "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" }
+pandas = { index = "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" }
+pyarrow = { index = "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" }
+scipy = { index = "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" }
+h5py = { index = "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" }
+bokeh = { git = "https://github.com/bokeh/bokeh" }
+# FIXME can't override distributed's conda dependencies with pip dependencies
+# partd = { git = "https://github.com/dask/partd" }
+# zict = { git = "https://github.com/dask/zict" }


This environment is brand new in CI. It already found genuine failures.

crusaderky · 2026-05-26T21:40:32Z

+test = "pytest"
+coverage = "coverage"
+coverage-clean = "rm -f .coverage* pytest.xml"
+test-ci = "pytest --cov=distributed --cov-report=xml --junit-xml=pytest.xml --runslow --leaks=fds,processes,threads"


Unlike dask/dask, this does not use pytest-xdist. Something to work on in a follow-up.

crusaderky · 2026-05-26T21:43:15Z

+          # Free-threading (WIP - tests don't pass yet)
+          # - os: ubuntu-latest
+          #   environment: py314t
+          #   task: test-ci
+          #   partition: ci1
+          # - os: ubuntu-latest
+          #   environment: py314t
+          #   task: test-ci
+          #   partition: not ci1


crusaderky · 2026-05-26T21:45:23Z

@dask/maintenance this is ready to go

crusaderky · 2026-05-26T21:47:08Z

The two failures in the nightly environment are genuine. They were undetected before.
I'll tackle them in a follow-up

FAILED distributed/shuffle/tests/test_shuffle.py::test_processing_chain[True] - DeprecationWarning: The 'generic' unit for NumPy timedelta is deprecated, and will raise an error in the future. This includes implicit conversion of bare integers (e.g. + 1).Please use a specific unit instead.
FAILED distributed/shuffle/tests/test_shuffle.py::test_processing_chain[False] - DeprecationWarning: The 'generic' unit for NumPy timedelta is deprecated, and will raise an error in the future. This includes implicit conversion of bare integers (e.g. + 1).Please use a specific unit instead.

crusaderky force-pushed the pixi branch 2 times, most recently from 27f7752 to 3f62c35 Compare May 26, 2026 11:05

crusaderky closed this May 26, 2026

crusaderky reopened this May 26, 2026

crusaderky mentioned this pull request May 26, 2026

uvloop tests #9278

Merged

crusaderky force-pushed the pixi branch 2 times, most recently from c9d1e43 to dac43ed Compare May 26, 2026 17:24

crusaderky changed the title ~~WIP Migrate to pixi~~ Migrate CI to pixi May 26, 2026

crusaderky marked this pull request as ready for review May 26, 2026 21:27

crusaderky requested a review from fjetter as a code owner May 26, 2026 21:27

crusaderky commented May 26, 2026

View reviewed changes

Migrate to pixi

cd9bc86

crusaderky force-pushed the pixi branch from dac43ed to cd9bc86 Compare May 26, 2026 21:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Migrate CI to pixi#9276

Migrate CI to pixi#9276
crusaderky wants to merge 1 commit into
dask:mainfrom
crusaderky:pixi

crusaderky commented May 25, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 26, 2026 •

edited

Loading

Uh oh!

crusaderky May 26, 2026

Uh oh!

crusaderky May 26, 2026

Uh oh!

crusaderky May 26, 2026

Uh oh!

crusaderky May 26, 2026

Uh oh!

crusaderky May 26, 2026

Uh oh!

crusaderky May 26, 2026 •

edited

Loading

Uh oh!

crusaderky May 26, 2026

Uh oh!

crusaderky May 26, 2026

Uh oh!

crusaderky commented May 26, 2026

Uh oh!

crusaderky commented May 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

crusaderky commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Functional changes

Follow-ups

Known issues

Uh oh!

github-actions Bot commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unit Test Results

Uh oh!

crusaderky May 26, 2026

Choose a reason for hiding this comment

Uh oh!

crusaderky May 26, 2026

Choose a reason for hiding this comment

Uh oh!

crusaderky May 26, 2026

Choose a reason for hiding this comment

Uh oh!

crusaderky May 26, 2026

Choose a reason for hiding this comment

Uh oh!

crusaderky May 26, 2026

Choose a reason for hiding this comment

Uh oh!

crusaderky May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

crusaderky May 26, 2026

Choose a reason for hiding this comment

Uh oh!

crusaderky May 26, 2026

Choose a reason for hiding this comment

Uh oh!

crusaderky commented May 26, 2026

Uh oh!

crusaderky commented May 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

crusaderky commented May 25, 2026 •

edited

Loading

github-actions Bot commented May 26, 2026 •

edited

Loading

crusaderky May 26, 2026 •

edited

Loading